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Abstract 



Effective project evaluation must rest on data which are comparable 
from one group or project to another and comparable across time, in order 
to measure changes in student performance accurately. Achievement related 
to individual goals of a project or set of projects oust be clearly 
observable . 

Usual classroom tests are not comparable across classes or over time, 
since each test io given only once, and only to one group. Usual project 
evaluation in urban schools is rendered almost invalid in some cases by 
high turnover in student population 1 the test is effectively nothing more 
than an immediate post-instruction test on the last few objectives taught. 
In addition, teat items are not usually tie i closely to stated objectives, 
especially when standardised testa arc used. The CAM model of evaluation 
overcomes these problems by giving comparable teste on all objectives 
throughout the course. Ccoputer-b^ed analysis and reporting make 
possible the handling of data for large projects. Several plans are 
possible for state-level evaluation of urban education programs. 
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I, Introduction 



The purpose of this report Is to present an evaluation technique 
called Comprehensive Achievement Monitoring (CAM), and compare it with 
two other common types of evaluation. These are usual clasaroota test- 
ing, and pretest-postteat project evaluation. These three evaluation 
mo dele will be related to the Guidelines for Nev? York St ate Urban E ducation 
Program , and also, to some suggested additions to the Guide) lnes . 

The CAM model is described in detail, and information and suggestions 
are given to aid in its implementation in evaluating urban education 
projec ts • 

Present guidelines for evaluation . The Guidelines for Sew York State 
Urban Education Program list some general goals for the evaluation of 
Urbnn Education programs, but do not specify these goals tn sufficient 
detail for Individual project and program directors to collect data use- 
ful at the state level . The Guidelines suggest that project evaluation designs 
should require: 

1) "data useful for decision-making at three levels--project, 
district, and state." 

2) "systematic and objective accumulation of information (on) 
strengths and weaknesses" of each project. 

3) data collection "prior to (the project's) Inception, during 
its operation, and at Its termination." 

<;) an "Index (or indices) of each project's effectiveness." 

S) information for making "decisions (on) continuing or modifying 
initial projects." 
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It is difficult to obtain an index of effectiveness, to observe an 
Increase or change in academic achievement, if the measures of achievement 
at one time are not dii'ectly comparable with the measures of achievement 
at another time (the notion of parallel criteria or parallel test measures). 
It Is therefore desirable to plan at the inception of a project a systematic 
pattern of frequent data collection, using similar and comparable instru- 
ments. Tne Guideline s call for information to be gathered throughout a 
project, although the specific types of data, and the actual schedule or 
design for the data collection, arc not enumerated in detail* However, 
the frequently-used posttest or pretest-po9ttest designs for project 
evaluation do not fulfill the implications of this goal. 

Suggested modifications for the G uideline s . In order to increase 
the usefulness of information for project and state level deci9ion-making, 
evaluation designs should require: 

1) cocroon instruments to evaluate similar projects. 

2) similar schedules of data collection, 

3 ) results reported in sufficient dccall to make Judgments about 
pacing and sequencing of instructional procedures. 

4) feedback of results during the project to improve instruction. 

5) a data bank of evaluation results upon which to base long-range 
policy decisions. 

6) financial data to combine with evaluation data to 
begin a cost-effectiveness analysis. 

Specific recommendations must be made at the state level to insure 
that projects systematically collect information which will fulfill the 
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requirements of evaluation. The information should include: 

1) estimates of performance related to the npeclfic 
objectives of the project. 

2) systematic and repeated pre-instruction testing 
of performance on each objective. 

3) immediate post-instruction testing of performance on 
each objective. 

4) retention tocting of performance on each objective. 

Care must be taken that the data gathered will allow a continuous, 
longitudinal appraisal, and that it will be collected in such a vay a3 
to keep to a minimum the effects of attrition. 




IV. Components of CAM 



CAM Is a procedure for testing achievement on every objective of a 
course, at frequent tost administrations throughout the course. At each 
test administration, performance on objectives not yet taught is pretested, 
performance on objectives Just taught is immediately posttested, and 
performance on objectives taught earlier in the course is measured for 
retention. Parallel test forms, comparable in difficulty and content, 
are all used at each test administration, but each student receives a 
particular form only once during the course. Each form typically has 
an item for each objective. Each item is uaed on only one test form. 

The function of a particular item changes in relation to the time at 
which its objective it taught. Testing may take place at regular intervals 
(e.g., every two weeks) or at the end of certain instructional units. 
Computer based analyses and reports arc available within a few days of 
data collection. 

Specification of objectives . The movt fundamental preparatory stop 
for the use of CAM is the specification of the objectives to be evaluated, 
in testable, behavioral terms. Objectives may be categorized according to 
•'•.noroua dimensions, and possibly organized into instructional units. 
Written objectives for a variety of closely related orojects or courses may 
be collated and pooled, l *. is then possible to identify and select for 
evaluation those objectives which arc common to several projects, and those 
that are unique to a project. Objectives are typically relaced to achieve- 
ment; however, CAM ia equally suited to measuring changes in attitudes or 
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perceptions. The pool of objectives is called an objective bank, and a 
computer program is available to handle the large amount of data Involved. 

Test items . The second step toward the use of CAM is the construction 
of test items. Every item is tied specifically to a single objective, and 
multiple items are constructed for each objective. All items, keyed by 
objectives, may be stored in a computerised item bank, ready for sampling 
or available for revision. 

Construction of test forms . The mmber of test forms, or monitors, 
mii9t at least equal the number of test administrations planned. Tests are 
made parallel in content by using the technique of stratified random 
sampling. Forms are also randomly comparable In difficulty, if an item 
analysis can be run (perhaps on a pretest or an earlier v» rslon of the 
course) for indices of difficulty and discrimination, the forms may be 
made more exactly comparable in difficulty. 

Monitors are intended to be short tests, perhaps ten to thirty items. 
Whether or not a single form covers all objectives for a course is a function 
of the proportion of objectives to iteos-per-form. It may be necessary to 
randomly sample ^without replacement) the objectives, before doing the same 
on the test items for each selected objective. This technique of sampling 
must insure that, across forms, all objectives are equally represented. The 
sane consideration holds when items-pet-form exceed the number of objectives* 
in this case, some objectives may be represented bv more than one Item on 
some forms. 

Student teat groups . Students arc divided into test groups in order 
to use all test forms at each administration. Test groups are best con- 
structed using random sampling of strata of students based on ability or 
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prior achievement In the subject. This assures that each group has a range 
of students which gives representativeness to the data for each test form. 

It Is most desirable, for several reasons, to Include every student In 
every test administration, and when set up this way, CAM has been found to 
be a satisfactory substitute for usual classroom testing. However, It Is 
possible to use only a ^ ample of the student population, especially if the 
number Involved In a project approaches one thousand or more. Many different 
sampling designs are possible; some are shown in Figure 20,1. Using the 
total student population in one test group is the design for the conventional 
project evaluation. Unequal-sized teat groups may sometimes be an administra- 
tive necessity. 

Test administrations . Test administrations may coincide with the 
completion of Instructional units, or they may be set at regular intervals 
throughout the course. The latter has advantages in terms of ease of 
administration, and comparability of results from similar courses taught r.c 
different schools. 

Appended package tests . It is possible to add a section to any monitor, 
and have the results incorporated with the rest of the CAM data. This feature 
lends flexibility in that, should a specific diagnostic test seem desirable 
at any point, the data can easily be assimilated. 

Data analysis and reporting . Output from the computer programs is as 
follows: 

For individual students 

After each administration: 

1) total score on that and all previous administrations. 

2) a graphic presentation of the above. 

3) a right-wrong indication for each item on the monitor, coded 
by the objective represented. 
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Fig. TM-20.1. Some patterns for samp- 
ling students. Circle represents student 
population. Shaded part represents sample 
tested at one administration, within which 
dlvieloRs indicate test groups. 
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At the end of the course: 

A) average scores, across all monitors taken, on Items categorized 
by use into three groups — pretest, immediate post-instruction 
and retention of varying lengths of time. 

For whole group or subgroups (e.g., one classroom; highest and lowest quartiles) 

After each administration: 

1) percent answered correctly out of all items across all monitor^, 
for each objective. 

Periodically, as desired (e.g., every 3-5 administrations): 

2) trend data, or achievement profiles, for total score and for 
each objective. 

At the end of the course: 

3) same as number A under individual students. 

A) item analysis (using whole group only) , treating each item 
in three separate wa';s, by its three functions — pretest, 
immediate post-instruction, and retention measure. 

Data are analyzed, and reports printed, by computer; results are 
usually available within a few days of each test administration. Data can 
be collapsed in various ways, to be most useful to students, teachers, 
project directors, or state evaluators. 



III. Usual Classroom Testing 



Description 

The usual classroom testing situation concludes the following sequence 
of events: first, a set of objectives is specified for a limited instruc- 

tional period, usually from one to four weeks; second, an instructional 
treatment is devised and administered to the students; and lastly, a test 
at the close of the instructional period is administered to measure the 
extent to which the objectives taught during that period have been achieved. 

Students' achievement on material taught during instructional period 
one is tested at test administration one. Achievement for period three is 
tested at administration three, and so on, throughout the course. 

There is usually a "final test" administered at the end of the course, 
for which there may be varying amounts of review offered. Sometimes major 
tests are administered at other times during the course, e.g. just before 
report cards are issued. 

Strengths 

Flexible weighting . There is great flexibility in the relative 
emphasis accorded various objectives during the year. Decisions may be made 
at any time; content may be added, dropped or modified. The testing is 
tailored to the content as the course progresses. 

Individual student testing . Usual classroom testing con yield 
diagnostic data on Individual student achievement, on the few specific 
objectives which have just been taught. 
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Teats related to objectives . Usual classroom testing may meet the 
criterion of close relationship between objectives and test items , when 
the school program is defined in behavioral objectives, and the teacher 
makes some effort to relate the items directly to the objectives. 

Limitations 

No pretesting . There is usually no pretest information on students* 
prior achievement on any objective. Teachers onually assume that student 
achievement is due solely to the instruction given them in class. Further- 
more, they do not know whether learning one objective has affected under- 
standing of another objective. Students may also have experiences in 
other courses, or outside of school, cither before or during a course, 
which contribute to their understanding of various objectives, whether 
or not they have been taught yet. 

No test of retention . There is no information on students' retention 
of objectives which have been taught earlier in the school year, except 
in the event of some sort of major test. At that test administration, the 
interval between time of instruction, and time of test-of- retention, is 
different for every objective taught. The interval may span almost a 
full school year, or be only a week or two. There is seldom any data 
attached to such test results about the date of instruction on a given 
objective . 

No comparison of student achievement over time. It is very difficult 
to compare students' achievement from one point in time to another, because 
at each test administration, an entirely different test is used; there is 
seldom any overlap in content, and the overall dfficulty can vary enormously 
from one test to another. The only possible comparison of achievement 
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from one time to another must use a student's rank order In his class. 
This still leaves no way to examine changes in a total class's achieve- 
ment over time. 
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IV. Project Evaluation 



Description 

A frequently used strategy for evaluating projects is to administer 
an extensive achievement te.3t at the conclusion of the project. This may 
consist of a test, or battery of tests, sometimes composed specifically 
for the project, but usually prepared and distributed commercially, e.g. 
standardized achievement tests. 

There is sometimes a pretest administered before the start of the 
project, which is either the same as the posttest, or an alternate 
form of it, but presines to measure the same objectives. 

Limitations 

Deficient immediat e post-instructional testing . In terms of im- 
mediate post-instruction achievement, the usual project evaluation 
measures only the objectives taught at the very end of the project in a 
way similar to usual classroom testing (l.e., immediately following the 
instructional treatment). This means that project directors do not have 
information on the direct effect of instruction immediately after students 
have been exposed to it. 

Tests of retention . The interval between the teaching of an objec- 
tive, and the end-of-course test, varies for each objective. Such inter- 
vals range from a week or two, to a full school year. Therefore, an 
estimate of achievement based only on a posttest is an aggregate of 
immediate post-instruction achievement, short-term retention, and long-term 
retention. This composite score may be made up of several subscores, 
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but such subscores still do not indicate much about the time interval since 
Instruction. 

No compariso n of scores . There is no need to discuss comparability 
of scores from one time to another if the testing is done at only one 
point in time. Pretest-post test problems are discussed below under sample 
attrition. 

Teat items not specific to objective . In posttests which are de- 
signed to cover an entire course at only one administration, there is 
great variation in the specificity with which test items have been matched 
to the objectives of the course. This problem is especially apparent when 
standardized achievement tests are used, where general subscores are roughly 
matched with the stated objectives of the project. When only standardized 
tests and materials are used in a post-project evaluation, there is a de- 

t 

finite lack of systematic inv'ormation about the achievement on specific 
objectives in the program. 

Inappropriate weighti ng. In giving one large posttest, especially 
a standardized test, the problem of weighting of objectives presents itself. 
A variety of objectives could be poorly measured while other objectives 
are heavily emphasized. It is likely that the intended pattern of em- 
phasis in the course will not be reflected in the evaluation instrument. 

Test not comprehensive . Not only will there be too little emphasis 
on certain objectives, but it is possible chat some objectives will not 
be measured at all. Lack of comprehensiveness in an evaluation technique 
is a serious shortcoming. 

Problems of samp le attrition . All of the above weaknesses in the usual 
project evaluation design are relatively unimportant when compared with the 
most serious problem of all: the turnover of students, which in urban 




schools Is extremely high; i.e., It sometimes equals 100JS of the enrollment. 
Those students who were pretested before the program, and received the early 
segments of instruction, are simply not there at the time of the posttest. 
Effectively, this reduces the hard data to a posttest on students still 
enrolled in the project during the final week, even if a pretest were ad- 
ministered. Therefore, the results may represent very little more than 
immediate post-instruction testing on the objectives taught just before 
the posttest. Pretest information, if it has been gathered, relates only 
to the incoming abilities of a sample of students roughly similar to that 
available for posttesting. The assumption is made that students coming 
into the project are similar to those leaving it, but the data cannot be 
used statistically in analyzing changes in student achievement, since change 
should only be measured for individual students or identical groups of 
students. A pretest-posttest design for evaluating projects does not seem 
to be reasonable for the types of programs usually encountered in the Urban 
Education Program. 

Advantages 

In the light of the serious limitations of posttest and pretest-posttest 
evaluation designs, it may be ironic to point out one seeming advantage, 
but in the interest of perspective, a single posttest (and perhaps pretest- 
posttest) does cost less than a more effective and complete evaluation 
system such as CAM. There is a minimum of clerical and administrative 
work needed in actually giving the test, and if a commercially available 
test is used, it may simply be purchased; no staff or time is needed to 
develop a test tailored to the objectives of the project. What little 
analysis on results can be done, is relatively easily accomplished. 



V. Capacity of CAM 



Advantages 

This section highlights the comprehensiveness, the precision, and the 
timeliness of information available from the CAM model, as contrasted with 
the usual classroom testing and project evaluations. It should be empha- 
sized that the validity of the estimates of group achievement available 
from CAM is comparable with, and in many respects superior to, that of the 
more familiar techniques. 

Specificity of objectives . Any project, no matter how it is to be evalu- 
ated, can call for a high degree of specificity of objectives; CAM, however, 
rigorously prescribes and requires such specificity. It is the base upon 
which the detailed testing, analysis and feedback of the program rest. 

Specificity of objectives allows similar projects to pool and match 
their objectives. What is common to all projects, or to several, is readily 
observable, and provides a meaningful, detailed comparison. Objectives 
unique to individual projects can pinpoint actual differences concretely and 
precisely. 

Test items tied to objectives . Each test item is constructed to measure 
achievement on a particular objective. Therefore, test data always relate 
to definite objectives, rather than aggregates of objectives; this allows 
evaluation procedures to be matched with specific goals of the project. In 
this respect CAM differs significantly from conventional project evaluations, 
where standardized materials are used, which have not been closely tied to 
the specific objectives for a project. 
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Modification of project s. Conventional project evaluation may provide 
some criteria upon which to base one kind of decision about an existing 
project: "drop it" or "continue it." These criteria are global rather 

than related to specific contributions of the project. Perhaps one of the 
most valuable characteristics of the comprehensive achievement monitoring 
model is that it is able to provide information upon which to make specific 
recommendations for retaining strong components of a project, and modifying 
weak ones. No project is as effective as possible, as set up at its incep- 
tion; therefore, a far more pertinent decision about it, now possible with 
the CAM model, is "drop" or "continue with these modifications." 

Data more valid . If there is time on a test for one question for an 
objective, then estimates of group achievement on that objective will be 
more valid if a variety of questions is used across the group, rather than 
the single question typical of both classroom tests and project evaluation. 

It is important to note that the increased validity and comprehensiveness 
calls for no sacrifice in the economy of data collection, since each student 
need still answer only one question. 

Pretest of all objectives . All objectives are pretested before any 
instruction has been given. First, it is Important to know whether students 
have already acquired information or skills from outside sources, so that the 
project need not lose students' interest by covering material that they can 
handle already. Secondly, an index of effectiveness must ultimately be an 
index related to change in student achievement, attitude or perception. In 
order to document change, it is necessary to have at least two comparable 
measurement of the same characteristic, taken at two different times. 

There is reason to continue pretesting on objectives to be taught later 
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in the project, because outside learning experiences, or interaction between 
material taught early In the project and that scheduled to be taught later, 
may both very reasonably cause changes In performance during the project. 

Thj.s may lead, to alterations, either In the sequence of instruction, or the 
amount of time spent on certain objectives. When the level of achievement 
rises on an objective not yet taught, it may be closely related to material 
Just taught, in which case. Instruction in the later-scheduled unit could 
be moved up to take pedagogical advantage of the relationship. Another pos- 
sibility is that, without changing the sequence, certain instructional units 
might be condensed, and the pace of Instruction stepped up. A single pre- 
courBe test, will not provide information for making the above decisions. 

Immedia te post-instructio n test . The usual classroom test covers only 
material just taught. CAM estimates of group achievement on just-taught 
objectives are comparable to those available from classroom testing. The 
numbers of students usually involved in projects makes it possible to test 
each objective with a substantial variety of items, without lengthening any 
one form of the test. 

Continual measur e of retention . Since objectives continue to be tested 
after they have been taught, throughout the rest of the course, there is a 
continual teat of retention. Intervals between "teach" and "test" times are 
of varying length, and can be matched for precise analysis. For example, it 
would be possible to measure retention spanning approximately six weeks on all 
material of a course except what is presented during the last month or so. 
Therefore, estimates of achievement can be systematically made for each of the 
instructional units after a specified interval. 
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One of the major indices of an effective program in urban education would 
be the retention of material learned during the project. This must be 
systematically and continuously pleasured, and the variable of time-of-retcntion 
must be included in any analyses of data. The usual classroom testing model 
does not permit this type of analysis because it coes not provide this type 
of data. 

Tailored revie w. Information on retention can have another side benefit: 
a possible saving of teaching time. Teachers often set aside class time before 
major tests for review. However, if it can be shown that students have not 
forgotten certain objectives, there is no reason for repeating the instruction. 

There are as yet no data to show whether the use of monitors, containing 
questions on previous materials, can actually aid in student retention. 

Achievement profile . There are comparable data on achievement for every 
test adminlat ration. This makes it possible to plot students* achievement 
on any given objective (or group of objectives) for the entire course. This 
plot, called an achievement profile, gives a graphic presentation of the 
changes in group achievement throughout the ourse. This achievement pro- 
file is a unique characteristic of the information available from the CAM 
mode) and is very useful in describing and reporting results of course and 
project research. 

Figure TM-20.2 presents hypothetical achievement profiles for five ob- 
jectives from a course. Brief contnonis below the graph give possible inter- 
pretations. It is obvious that achievement profiles provide a wealth of 
information, at whatever point in the course they are drawn. On the protest 
in the loregoing example, all objectives except nusber 2 show achievement at 
the chance level, or about 201 (five-option multiple-choice items). Several 
decisions could have been made after test administration one: 





(coincide with conclusion of instruction unit) 

ObJ. 1» taught, but students did not learn; with rapid 

feedback, could be corrected with change (n Instruction. 

ObJ. 2t previously known and not taught; without pre- 
test, this looks like student learning. 

*“’““** ObJ. 3: taught and learned, but forgotten 

* ObJ. 4: well taught 

* i obJ. 0 t related tc objective 3, because achieve- 

ment increases when 3 is taught. 

Figure TH-20.2. Hypothetical achievement profiles of group achieve 
ment on five objectives. 
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1) Objective 1 was not learned— re teach it in some other way; 2) 

Objective 2 has tested high on both the protest and test administration 1— 
it would be safe to skip instruction in this objective. After teat admin- 
istration 5, two other decisions might havo been made: 1) Achievement on 

Objective 3 seems to be slipping — review is needed, preferably soon; 2) 
Objective 8 seems closely related to Objective i— perhaps it should be taught 
now Instead of later. 

Other variations of achievement profiles are available. Figure TM-20.3 
shows total test scores averaged by quar tiles of the class. Other sub- 
groupings of students are possible. Any desired subgroupings of students is 
possible. Figure Ht-20.4 shows the profiles of four students (total scores) 
across the semester. The computer programs are general so that profiles for 
any combination of items for any subgroup of students may be obtained. 

Relatlvi i mmunity to attrition . To gauge the effectiveness of a project, 
change in student achievement must be documented; measurement must be avail- 
able for an objective both before and after it is taught, and the measurement 
must be on the same students. The usual project evaluation measures achieve- 
ment, at best, only at the beginning and end of a project. The rate of turn- 
over of student population in urban schools is notoriously high. However, 
the unique capability of the CAM model allows comparable measures 
of achievement at multiple points in time. The shorter the Interval between 
tests, the greater the likelihood that students enrolled in the project 
at test administration A, are still enrolled in the project at test adntnlstra 
tion B, Much data can be salvaged through this technique, but it is 
irrevocably lost when the usual pattern of project evaluation is followed, 
fhis is illustrated in Table 1M-20.J. 
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Figure 1M-20.3. Achieveaent profiles of quArtiles of a class by test adninlstra- 
tion. (Percent correct for each period la the tsean percent correct of that period 
and the tvo adjacent periods.) Students vere divided by their scores at Adninistration 
Intervals were about two weeks long. Administrations 1 and 16 were pretest aod poet- 
test, which were taken by every student* The othe* adsinistrat ions involved a 30X 
eaaple of students. 
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Figure TM-20.4. Achievement pro. ilea of four students; 
totel tsftt score by test administration. Intervals were about 
tvo weeks long. Adainlstrations 1 end 16 were pretest sod post- 
test, which were taken by every student. The other administra- 
tions involved s 301 saeple of students. 
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TABLE TH-20.1 
Evaluation Cost Analyols: 

Attrition of Hypotehtical Student Sample 
as Related to Percent of Evaluation Data Retained 



Group 
Entering 
After * 


Enrollment 


®>id end mid end 

September November January April June 


September 

November 

January 

April 


WOO 800 700 600 400 

200 175 150 100 

125 110 70 

140 90 

— 340 



Evaluation 

Design 


Percent of Data Usable 


For Period 


Percent 


Pretest-Posttest 


Sept ember* June 


40 


CAM 


September-November 


80 




Noveobe r- J anuary 


88 




January -April 


86 




April-June 


66 


1 


1 CAM AVERAGE 


80 



♦First figure in each row represents entering students* subsequent figures 
in that row represent students in that group atill in school, for whom 
are useful. 
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Continuous data available . Data are available from every test admin- 
istration. It is possible to look at group achievement on a single objective, 
groups of objectives, or total content of a course, though this last is 
generally less useful. Data can be summarized in a variety of ways, through 
the use of selected computer programs now available. Desired data are 
always available within a few days for decision-making; it is not necessary 
to wait weeks or months for meaningful analyseo. Many evaluation systems 
are not able to analyze and report results with sufficient speed and organ- 
ization to aakc the information most useful to its recipients. Analyses 
can be tailor-made for project directors or state evaluators. 

One economic advantage of periodic feedback is that a project need 
not continue to its end to discover, after all funds are spent, that the goals 
of the project have not been accomplished. Modifications can be made in the 
program if student performance does not move in the expected direction. 

Indices of effectiveness . One of the most pressing problems in the 
relative evaluation of projects is comparability of data from one project to 
another. Such a measure might be called an index of effectiveness. 

For decision-making at the state level, where several similar projects 
may be competing for continued support, courses must be compared on whether 
students achieve the behavioral objectives which they have in common. Since 
common objectives, and matching test items, can be selected from a large 
pool, data from several projects can be compared with precision. It is 
important to note that items (or objectives) that are different from one 
project to the next may have different levels of difficulty, and therefore 
comparison of projects using scores on these items may be distorted. 

Project CAM has a program that calculates for individual students, by 
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objective, scores for items wh^ch were pretest items, items which were 
immediate posttest, and for items which were retention measures (see Table 
TM-20.8; for more technical data sec 111-6, TM-11, Til- 12) . If monitors 
were either wholly or largely the same for several projects, it would be 
relatively easy to compare projects on these measures. Additional measures 
of effectiveness might include analyses of group learning curves, or more 
formal trend analysis or time scries regression. These measures could be 
applied tc achievement data, attitudinal data, attendance data and/or social 
(process) data. 

One further step in analysing effectiveness of projects, once a measure 
of effectiveness is established, would be to analyze effectiveness in re- 
lation to financial information on each project. Rudimentary cost-benefit 
analyses would shew which projects having comparable objectives were most 
effective p er dollar . This would seem to be the ultimate purpose of project 
evaluation. 

Important Considerations in Use of CAM 

Good items needed . The development of an item pool for CAM takes som:> 
special skills and considerable staff time. As in all large evaluation 
projects, items must be of high quality; with CAM, there must -Iso be more 
items. Ideally, an item analysis should bu run, to insure that all monitor 
forms are of equal difficulty. In the absence of such an analysis (and 
there often is not time or opportunity for it before starting a program), 
it is doubly important that the staff assigned to item construction be 
skilled and experienced. Forms will be randomly equivalent in difficulty. 

Staff needs . In addition to the initial construction of tese Items, 
other components of CAM also need th? attention of highly skilled staff 
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members. Items must be selected for each specific project; monitor forms 
must be carefully devised with proper sampling procedures; there may be 
student sampling procedures to accomplish; interpretation of test data 
must be dono ccrrectly; analyses and summaries call for certain technical 
skill. 

In addition to academic and technical staff needs, there is some cler- 
ical work Involved in keeping track of monitor forms, students, administra- 
tions, and results. 

Sampling problems . When it has been decided, for reasons of economy, 
to sample the student population rather than test everyone at every adminis- 
tration, certain problems can arise. 

Students are accustomed to feedback from most tests they take; they nay 
become indifferent to a series of tests which seem sporadic and yield no 
reinforcement. Also, a student may be resistant to taking a test if he 
knows that some of his friends are not. 

When CAM is set up to test comprehensively (every student on every 
objective at every administration) it has been found to be an adequate 
substitute for classroom testing. Obviously, if students are being sampled, 
classroom tests for diagnosing individual student achievement must be given 
in tandem with the monitors. 

Attrition problems . There is no way to solve completely the problems 
of student sample attrition. Achievement on specific objectives will show 
up, but if there is a rapid turnover in the student population, total scores 
nay rise very little. Some of the information about transfer of learning 
from one objective to another may also be distorted. 

Low total scores . Total scores for a project oust be interpreted care- 
fully. If scores are lew, several factors may be at work. Attrition problems 
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can account for "learning and forgetting" achievement profiles. Another 
problem may be the overall difficulty of items, vhich of course are not modi- 
fied during the year. Careful analysis can obviate these difficulties; 
it is important that incorrect interpretation not be the cause of unfavor- 
able comparison to other projects. 

Summary Comparison of Three E valuation Models 

The amount and quality of information available from the three models 
of evaluation described above will serve to summarise the characteristics of 
each. 

Comparison of matrices (amount of information ) . CAM yields more infor- 
mation than either the usual classroom testing or conventional project eval- 
uation. The pattern of data resulting from each model may be fitted into a 
matrix, in which the rows indicate all the objectives or instructional units 
of the course, and the columns represent the possible test administrations 
during the entire project. A cell of the matrix which is filled in, represents 
en estimate of achievement for that objective or unit, at that test adminis- 
tration. 

The usual classroom testing pattern is illustrated in Table TM-20.2. The 
diagonal line of X's represents the series of separate tests on each 
instructional unit, each given at separate administrations. The column of 
X's at the last administration indicates a final teat, presumably covering 
ell the units of the course. 

Table TM-20.3 illustrates graphically the lack of information available 
from the usual pretest-posttest project evaluation. This illustration makes 
the assumption, not necessarily well-founded, that a single test does In fact 
provide information about every Instructional unit. 




TABLE TM-20.2 



Usual Classroom Testing: Estimates of Achievements 



Available 


for a Group of Students by Unit 
Test Administration 


and 








Time 




Unit 


1 


2 3 4 




1 


X 




X 


2 




X 


X 


3 




X 


X 


4 




X 


X 


• 

• 




X 


X 


U 






X 



TABLE TK-20.3 

Pretest-Postteat Project Evaluation: Estimates of 

Achievement Available for a Group of Students 
by Unit and Test Administration. 



Tine 



Unit 1 2 3 4 ^ T 

IX X 

2 X X 

3 X X 

AX X 

. X X 

• 
s 

U X X 
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It is readily apparent in Table Ttt-20.4 that CAM makes available 
data on group achievement for all of the objectives specified for c course, 
at each time of testing. This comprehensiveness of the data provides 
the necessary information for the varioty of purposes discussed earlier in 
this section. It is easy to see how CAM contrasts with the other models of 
testing, where information lo generally available either on a few of the 
objectives, or as a composite score for all objectives, at a single time. 

Quality comparison . Table TH-20.5 displays seven types of information, 
and estimates their quality as provided by each of the three models. 

Conventional project evaluation is fair to poor on all of the dimensions 
described. These shortcomings are Inherent in the use of single tests at 
one, or perhaps two, points in time. A single test long enough to provide 
detailed information about student performanco on a large number of objectives 
is fatiguing and therefore less valid than short tests. One long test 
excludes systematic pretest, immediate pon-inat ruction, and detailed reten- 
tion information. Attrition takes a heavy toll of a pretest sample. Feedback 
is limited to a post-mortem on the project's strengths and weaknesses. 

Usual classroom testing provides for the measurement of performance in 
specific objectives on an immediate post-instruction basis. By repeated (eat- 
ing, the effects of attrition may be minimised, If usual classroom testing 
data were collected acroas similar projects after similar objectives had 
been taught, extensive information would be available for comparing projects. 
However, an accurate comparison of projects must also Include pretest and 
retention information. The former is used to adjust for Incoming aptitude 
and achievement differences in students, and the latter for long-term retention, 
or payoff of the project. Neither of these la specifically available from 




TABLE Tti-20.4 



Comprehensive Achievement Monitoring Evaluation: 
Estimates of Achievement Available for a Group 
of Students by Unit and Test Administration 
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TABLE TM-20.5 



Quality of Information 
Available from Three Evaluation Models 







Model 




Information 


Usual 

classroom 

testing 


Usual 

project 

evaluation 


Comprehensive 

Achievement 

Monitoring 


Pretest of 
objectives 


A 


A 


AA* 


EvaJ uation 
specific to 
objectives 


AAA 


AA 


AA* 


Immediate post- 
instruction 
testing 


AAA 


AA 


AAA 


Evaluation of 
retention of 
objectives 


A 


AA 


AAA 


Comparability 
across time 


A 


A 


A A* 


Achievement 

profiles 






AAA A 


Continuous 

feedback 


AA 


A 


A A A A 


Immunity to 
sample attrition 


AAA 


A 


A* AA 



NOTE: 



Quality of information rated as excellent (****), good (***), 
fair (**), poor (*), and not available (blank). 



- 31 - 




«• 32 - 



classroom testing. Feedback occurs frequently during the project, but 
provides information about only one instructional unit at a time. 

Comprehensive Achievement Monitoring provides information for project 
or state evaluation comparable, or superior, to the other evaluation models. 
Its superiority lies in the areas of particular importance to project evalua- 
tion: systematic pretests and measures of retention of objectives. Attrition 

which can easily invalidate the results of an evaluation of urban education 
programs, is accommodated in an automatic way. Feedback can be provided 
continuously and comprehensively (as called for in the Guidelines ) so that 
the projects can be critiqued and adjustments made before their end. 




VI. Resources of Project CAM 



Staff 

Project CAM has been involved in n variety of courses and grade levels 
in the process, a staff hat, been assembled with a valuable range of skills 
in using CAM procedures. 

Experience in urban education . Several CAM staff members have initi- 
ated, or worked with, educational projects for the disadvantaged. They 
have worked with Headstart programs, occupational guidance in a ghetto, a 
"drop-out prevention" project, and individualized Instruction in inner 
city high schools funded with Title I monies. The combination of project 
activities in another context, and participation in CAM, has allowed these 
staff members to develop an expertise well-suited to urban education evalu- 
ation. 

Perhaps the most difficult aspect of evaluating urban education 
projects lies in establishing rapport with project teachers and students. 
They must be approached by competent and sensitive evaluators; and the 
validity of the data collected can be seriously compromised by indifference 
or lack of understanding on the part of teachers or students. The ideal 
situation would be to have a few highly trained, skillful and sympathetic 
evaluators, who could go into schools involved in a project. They could 
explain the evaluation system, work out specific problems that -‘ight 
arise, and train one or several local staff members to administer the 
monitors correctly. 
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Experience in design and analysis , Fvery project has characteristics 
which are unique to it: objectives, pacing or sequencing of instruction, 

class organization of students, etc. Each of the courses using CAM has 
had a different structure: teachers singly or in teams; traditional, 

individually paced, or individually prescribed instruction, Usin& the 
basic CAM design, a specific evaluation design was developed to suit each 
course. Statistical analyses were performed to provide each teacher with 
the maximum amount of information concerning the special features of his 
course. Members of the CAM staff have demonstrated their flexibility 
in accommodating variations in course (or project) requirements. 

Table TM-20.6 shows the range of courses for which CAM has been used, 
or is presently being planned. Table TM-20,7 gives some detail for courses 
where CAM is in current use. 

Computer Software 

In order to be most useful to project directors and state level deci- 
cion makers, the steps of an evaluation must be accurate, flexible and timely. 
CAM procedures are capable of such accuracy and speed, because of the devel- 
opment of related computer software, i.e., computer programs, to handle the 
large amount of data involved. A summary of available programs follows. 

Objective bank . This program stores the text of objectives, and allows 
them to he revised. Objectives may be categorized in a variety of dimensions 
and may be retrieved through any category. The program will print lists of 
objectives , 

Item bank . Items are coded by their objectives ?nd/or other indicators* 
they can be stored (with correct answer keyed), edited, selected at random 
within objectives, and printed in sets to produce randomly parallel test 
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TABLE TM-20.6 



Subject and Grade 


Levels for 


which 


CAM is 


Operational or Planned 


Subject 


G 


rad 


e 1 


eve 


1 


Pre-high 

school 


9 


10 


ii 


Post- high 

12 


Science 


P 


0 


0 


p 


0 


Mathematics 


P 


0 


0 


0 


0 0 


English 






0 






History 


P 




0 


0 


0 


Vocational 




P 


p 


p 


P 0 



- 35 - 




TAELE TM-ZC.7 









1 


























CM 


























M 


o 


o 


m 


CJ 


O'* 


sr 


st 




o 






0) 


a> 


CM 


CM 


rH 


CM 




rH 


CM 


rH 


CO 






M 


























JH 


























M 


i 

o 
























CJ 


SM 


O 


O 


CA 


o 


<r 


m 


CM 


as 


o 






-9 


U-i 


H 


rH 




rH 


H 


rH 


H 




rH 






§ 

is 


O 




















a\ 






Pn 




















SO 






O 




















f 






0 




















CO 




w> 


<D 




















vO 




c 


0 


Ps CA 


Ps CA 


Ps CA 


PS W 


Ps 


Ps co 


Ps W 


pS CA 


Ps ca 


Os 




•H 


O' 






k 04 


U M 


U 04 


U 04 


M 40 


C* 44 


U 44 


rH 




M 


a> 


<1' £ 


<U > 


0) > 


G) £ 


a> d> 


a) £ 


0) ^ 


G> * 


a) > 






W 


u 


> 


> 


> 


> 


> J 


> 


> 


> 


> 


S' 


ft) 


CJ 

H 


«M 


W LO 


PC -V 


w s* 


w <* 


u\ £ 


M CM 


W 


W CM 


W CO 


-4* 


4J 
























u 


ctf 


























« 
























PS 


























03 


M 


























a 
























*0 


d 


O 


CJ 








• 












cl 


*> 


rn 


H 




















n 


<y 


M 


•H 




















o 


rl 


3 


(A 


o 


o 


o 


o 


o 


o 


o 


O 


o 


4J 


<D 


O 




in 


sf 


•sf 


rH 


co 


CO 


m 


in 


m 


*H 

a 




a 




rH 

♦ 


rH 


rH 


rH 


rH 


CM 


rH 


rH 


rH 



w 

o 

CA 

u 

3 

o 



0 

o 

•H 
IH u 

o o 

3 

'U M 
O 4J 

02 w 
fl 

<y *h 



*T> H 

Q) 

N 

*h a 

H O 
d *H 
3 4J 
T 3 O 
•H 3 
> M 

•r{ 4J 

•a w 

a a 





rH 




rH 


rH 






d 




d 


d 






0 




d 


C 






O 




o 


o 






*H 






■rH 






M 




M 


M 






•H 




*H 


•H 




• 




• 


"O 


T> 


• 


M 


d 


M 


d 


d 


M 


• 


M 


• 


M 


M 


• 


IH 


H 


M 


H 


H 


H 



m a) 
a) 


X 


02 


02 














u d 


4J 


M 


4J 








M 






M M 


O 


O 


o 








rH 






d co 


rH 


rH 


rH 


02 


02 




rH 


02 




£ 








4J 


4J 


o: 




M 


02 




"* 


r 




O 


rH 


u 


m 


rH 


u 


m a 


X 


Ps 


P^» 


rH 


rH 


0\ 


Ps 


rH 




o d 


w 


M 


CO 








M 


41 


a> 


•rH 


O 


o 


• 


r 


r 


O 


^ 4J 


rn 


*r-> 


rH 


M 


H 


02 


02 


02 


4J 


02 CM 


rC 


X 


CC 


UJ 


o 


4-» 


Hi 


4J 


CA 


4-J rH 


M 


3 


0 


vH 




d 


r'-J 


d 


•H 


d 

; ^ 


d 


to 


w 


PC 




•r » 


>1 


PC 


% • 
/- 



<y 


a 


o 


rH 


O 


o 


CM 


o 


CM 


in 


(A 


rH 


rH 


CM 


rH 


CM 


o 




Sf 


o 


M 

H 


rH 


CM 


CO 


<r 


•sfr 




CM 


<T 


Cvl 


3 


PU 


§ 


02 

n 


PC 

n 


to 

PC 


n 


S 2 


0. 


^ * 



- 36 - 



O 

ERIC 



- 37 - 



forms. CAM procedures may call for 600 Items to be distributed across 
40 test forms, for one of numerous related projects; the computer program 
can accomodate many thousands of items, belonging to any number of 
objectives. More information on the item bank program is available in 
Technical Memorandum 14 , and in Gorth and Grayson (1969) . 

Data bank . This is a "tope building" program; i.e., it continually 
adds information in the correct place on one tape, which is the data storage 
vehicle. The program inputs, edits, collates and systematically stores 
information on each student, and the instruction on each objective; in 
addition, for every monitor of the set, each item is noted with its correct 
answer, and categorized in up to fifteen dimensions (e.g., Bloom's taxonomy, 
1956), one of which is the objective to which the item is tied. Data on 
each student include: 

1) his name awj. available background information, personal and academic. 



2) the sequence of t£»ts he is assigned, which is independent of 
other students' sequences. 

3) the date he completes instruction on each objective. 

4) at each administration, the date, test form and specific responses 
he gave. 

It is evident that no data are lost, and the format of this program 
allows flexibility: instruction may be individually paced and/or prescribed 

if desired, and students may be individually scheduled for testing. For 
more information, see Technical Memorandum _7. 
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Test data analysis and reporting . As soon as test data are put into 
the data bank, they are available for analysis, along with all the previously 
stored data. This program, therefore, calls on the data bank for its raw 
material. Reporting includes: 1) group achievement on each objective 

over time; 2) total test score for the current and all previous administra- 
tions, for individual students; 3) right-wrong information on each item, 
keyed by objective, taken by an individual student of that administration. 
Figures 1M-20.5, TM-20.6, and TM-20.7 show sample outputs of this program. 

Achievement profiles . This program provides a detailed analysis of 
performance, for 1) groups of students, e.g., group achievement on a single 
objective, or 2) groups of items, e.g., trend in total score for each 
student, or 3) combinations of 1) and 2). It is a graphic presentation of 
achievement across time. Sample output can be found in Figure TM-20.8, 
and more information is available in Technical Memorandum 12 , and Gorth, 
Grayson and Stroud (1969). 

Item analysis . Not every student receives every 1 a after the 
corresponding objective has been taught, and there is no one test score 
which can provide a valid criterion against which to measure the diffi- 
culty of a given item; therefore, the usual procedures for obtaining 
difficulty and discrimination indices are not applicable to CAM data. A new 
procedure was developed (Technical Memorandum 6; Lindeman, Gorth, and Allen, 
in press) which allows three separate treatments of each ite.n, by its 
function: pretest, immediate post-instruction or measure of retention. 

Implementation by a computer program is discussed in Technical Memoran - 
dum 11 . Sample output is abstracted in Tables TM-20.8 and Ttf-20.9. 
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Figure TM-20.5. Sample computer output of group achievement 
on instructional units for administration eleven and all previous 
administrations. Course is 11th grade trigonometry; weekly tests 
were more frequent than completion of units. Pretesting indicated 
little prior knowledge of subject. 
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Figure TM-20.6. Sample computer output at each test adminitration: 
students' total scores for administration 10 and all previous administra- 
tions . 
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Figure TM-20.7. Samples of individual student reports distributed after each 
test administration. PC is performance criterion number (objective number; a 
student may refer to his list of objectives). A is answer: + (right); - (wrong). 

PD is period (test administration) and SC is score. 
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Figure TM-20.8. Staple 
computer output evslleble 
when desired] profile for 
group (or subgroup) »chieve~ 
•ent on one objective or one 
instructional unit. 
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TABLE TM-20.0 



Item Analysis: Criterion Scores by Tine 

Intervals (days measured from completion 
of instruction). Appropriate items col- 
lapsed across intervals. 



Student 




Interval 




-150 / -15 
(Pre-) 


0 / 60 
(Post) 


61 / IjO 
(Retention) 


18 


03 


38 


59 


63 


01 


60 


60 


79 


14 


82 


84 


109 


07 


27 
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TABLE TM-20.9 

Item Analysis: Difficulty Indices 

by Tine Intervals (days measured from 
completion of instruction). Appropriate 
students collapsed across intervals. 







Interval 




Item 


-150 / -15 


0/60 


61 / 150 




(Pre) 


(Post) 


(Retention) 


Difficulty 


33 


.07 


.65 


.94 


51 


.12 


.67 


.56 


119 


.09 


.32 


.69 


174 


.09 


.29 


.16 


Discrimination 


33 


-.42 


.48 


-.14 


51 


-.15 


.43 


.72 


119 


-.12 


.83 


.48 


174 


.06 


.14 


-.13 
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VII. Implementation 



Three Alternatives 

CAM would prove valuable aa one major form of evaluation of urban edu- 
cation projects. However, there could be considerable variation in the 

( 

extent to which the state departnent (or other agency) provides expertise 
and assistance in evaluation. The degree of Involvement of an agency 
external to a project is, in general, positively related to the quality 
and generalltsblllty of the data collected. It would specifically affect 
comparability measure of effectiveness between projects concerned with 
the same objectives. Three alternatives are suggested below (see Table 
TM-20.9) for allocating authority for the various steps in the CAM procedure. 

Handbook . Central to any plan to implement CAM would be the preparation 
of a handbook. This would provide a detailed Jescrlptlon of the CAM 
procedure and its specific application to urban education projects. An example 
of this type of handbook is the Assessment and Evalutatlon- - Iltle I-- ESEA , 
which was developed by the Nev York {State Education Department. 

_I . Project autonom y. With the aid of the handbook, a project 
director could design a CAM monitoring procedure for his project, including 
objectives, actual test items, ■ .d monitoring schedule. This preparation 
could be required as part of ne proposal, or before funds were released, 
and would be critiqued at f* ** State level. Individual projects would be 
responsible for the colled ->n and analysis of data, and the reporting of 
results. 




44 



TABLE TM-20.10 



Alternatives for State Level Control 
of Components of CAM Evalution 



Component 


I 


Alternative 

11 


III 


Cam handbook 


X 


X 


X 


Objectives 




X 


X 


Instruments 




X 


X 


Design 






X 


Collecting data 






X 


Analysis of data 






X 


Reporting 






X 
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One benefit of such intimate involvement with the evaluation is the 
increased likelihood that project directors would correct weaknesses as 
they became apparent. Decisions about pacing and sequencing would rest on 
the working knowledge gained from defining objectives and evaluation 
instrument, and from following a systematic procedure during the year. 

A potentially serious limitation of this alternative is the variation 
across projects in the quality and uniformity of instruments, and in the 
quality and uniformity of collection and anlysis of the data. 

II : State specified objectives and items. A State level agency would 

be responsible not only for the handbook, but for clearly articulating the 
objectives and associated items for a project or aeries of related projects. 
This would probably Involve the use of a computerize J item bank, to facili- 
tate the distribution of the items in the forn of a set of CAM monitors. 

Since this process takes skilled staff and time, materials would be developed 
gradually; for the moat pt.y-off, projects affecting the largest number of 
students would be chosen first. 

Possible advantages of alternative II over I include the probability 
that cbjectivea will be better clarified and items more valid. Also, 
uniformity of items from project to project for evaluating similar objectives 
allows clearer comparisons of project success in achieving cbjectlves. 

Part of the .validity and comparability of infornatic..-. is a function of 
the reliability and uniformity of the design, data collection, analysis, 
and reporting. Since these elements ere left in the hands of the individual 
project's personnel, a relative comparison of projects is less valid. It 
oust be noted, however, that validity of data collection is very much a 
function of the training and carefulness of the person actually administering 
the monitors, given a good evaluation design. 




Ill . State co ntrol over all components . Alternative III calls for a 
comprehensive evaluation to be made of each project, with administrative 
responsibility separate from the project. The evaluation agency would pre- 
pare and operate all major components of the evaluation, and would continu- 
ously report results to both the project and the State Education Department. 
Specifically, the ap.ency would write a CA ' handbook, assist the project in 
articulating its objectives, develop items, specify the design , collect and 
analyse data, and report results. 

The resources needed for the evaluation agency would be developed 
gradually over a one-and-a-half to a five-year period, depending or. the 
level of financial support. The development would begin with a sample of 
projects with high priorities e.g. reading for non-English speaking 
students. Many of the resources developed for the reading evaluation 
(e.g. staff, computer programs, data collection techniques) would be 
directly usable with othr project*. Only additional objectives, items, 
and staff would then need to be developed. 

Budget and Timetable 

Table Tll-20.11 presents some approximate costs for the three alternatives 
presented above. Since Alternative III encompasses the first two alternatives, 
the table presents the costs cumulatively. The estimates are itemized, to 
make it posaible to adjust them to mote probable rates within the Education 
Department. It should be noted that these costs are based on ten monitor 
administrations in fifteen projects of the same type. 

For a one-year project, it might be feasible to think in terms of 
five monitor administrations: one each quarter and one at the beginning of 

the year. However, it would be much more difficult to correct the plan of a 




TABLE TH-20.11 



Cost Estimates for Fifteen Projects of Same Type 







Estimate Procedure 




Alternative 


Category 


Itemization 


Cost 



I 



II 



Handbook 



$ 15 000. 8 1 



2 writers, 2 months $ 5 000. 
secretary, 3 months 2 000. 
printing, 10 000 cps. 5 000. 
Overhead 



Objectives and Items 



40 000. a,b 



2 wk. wkshp. exons. 5 000. b 

2 people/proj. £ $400. 12 000.? 

1 administrator, 3 mos. 5 000. b 

2 curric. consult., 2 mos. 2 500.° 

3 curric. consult., 2 wks. 1 500. b 

2 secretaries, 3 mos. 3 000. 

overhead, duplicating, 

telephone, etc. 



Computer based test forms 



10 000 . 



keypunching, 2 items/5 min. 6 



$4. /hr.; 160 hrs. for 
4000 iteas c 800.® * b 

computer time (development, 

print monitors on mimeo) 2 000. 

sampling programmer 1 000.® 

item-bank programmer 2 500.® 

running technician 1 000.® 

Evaluation design 3 000. b 

$200. /project 



(Each additional project of same type) 
(Projects of another type; per project) 



(1 000 .) 

(5 000.) 
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(Table TH-20.11, Continued) 



III Revise and further develop conputer 



programs for data bank, 

analysis, and reporting 25 000. a 

computer tine; 1/2 hrs./da.; 

5 da./wk. ; 12 wke. 

(30 hrs. @ $300.) 10 000. 

two programmers for 12 wks. 8 000. 

Data collection and anlysle 40 000. ^ 

15 projects; 1 test/4 vks. 

■ 1 test/proj . /day 

1 data collector - full time 12 000. 

Training (1 bo.) - Trainer 

Other expenses 1 500. 

Punch data (e.g. Digitek) 

500 Ss/proj./lO tests 9 Sc 4 000. 

Analysis & reporting! 

15 tain. /proj. /testing 
@ $300. /hr. 10 500. 

Reports to projects 300. 

nailing $1. /testing 
handling $l./tostlng 
State analysis of data 

1/2 da. /test/proj. »75 uays 
0 $50. 3 750. 



Sowary 

I Handbook $ 15 000. 



11 


Project autonomy 


70 000. 




Handbook 


$ 15 000. 




Objectives and itens 


40 000. 




Tdat forms 


10 000. 




Evaluation design 


3 000. 


III 


State control over all components 


135 000. 




Alternative It 


$ 70 000. 




Computer programs 


25 000. 




Collection and Analysis 


40 000. 



Notet Costs for alternatives are cumulative (see Sumsry). 

Totals are rout*, figures, not exact susaatlons. 
a one tins coats 
b reseating coats 

c 1000 itaaa In common, 200 unique items for each project, 
d depends aomevhet on number of projects in the type 



<0 
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course if half a semester were allowed to go by without feedback. The chief 
advantage of the use of CAM for state level evaluation is the data that are 
fed back to the projects for their own adjustment of strategy before the 
course is over. The cost estimates reflect this philosophy. 

Table TM-20.12 presents the tightest practical schedule for executing 
each component of CAM. The following points must be held in mind while 
interpreting the timetable. 

1) The handbook is scheduled here to be completed before the construc- 
tion of objectives and items. However, this is not necessary; state department 
personnel have many resources for constructing a curriculum in behavioral 
terms, and writing valid items for measuring specific objectives. The hand- 
book should be completed in tine to aid the writers of incoming proposals. 

It is obviously more important for those who may construct curricula and 
tests without the direct supervision of the state department (Alternative I). 

2) Time allotted for each task minimum, and assumes fulltime, 
efficient attention; each task is scheduled at the latest possslble time 
it could be done* Heedless to say, it la advisable to leave extra time 
for unforeseen delays. Time allowances must also be adjusted if the 
personnel involved are working on other assignments simultaneously. Some 
taske may be completed sooner if more personnel are available. 

3) The timetable la related only to the taske which need to be com- 
pleted before actually starting to use CAM; it is independent of who dot's 
the work; l.e., it is not directly related to the Altemetlvee suggested 
above. However, it is inevitable that coordinating the efforts of more 
people on more levels will take more time. 
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Practical Considerations 

Planning CAM early . To bo most effective, CA must he planned when 
a course is planned, in fact, CA’ f is Inseparable from the content ami 
structure of a course, lime for preparation funds % and staff must be 
integral parts of the evnJuation planned in a proposal. It oould he advisable 
to require such a plan as part of every oromo n) , bother nr not it vms in 
be modified later with the help of State consul t ants* 

Coot of CiVs If a snail expenditure on evaluation doesn't yield 
reliable data, it is very expensive evaluation. If five to ten percent 
of project funds were allotted to :;ell-desJpned evaluation, it could alio*; 
total dollars spent in education to have substantially nore impact. There 
are two major observations to make about the cost of C,V*. 1) If the State 

were to coordinate personnel from similar projects for objective and item 
writing, the additional cost for ( A ! would represent a fraction of the 
funds already allocated to traditional evaluation. 2) The basic preparation 
for Ck" is the writing of objectives in clear, testable, behavioral term.-, 
and constructing items to measure specific objectives nbviouslv, just 
preparing to use C.V< will vastly Improve the quality of a project, quite 
apart from actual data collection or reporting results* It ic questionable 
whether this expense should be charred to "evaluation," or to "curriculur 
construction." If the curriculum has been wli specified, then setting up 
a C A?? evaluation procedure is relatively ea*v. iTien the benefits of feed- 
back to both students and teachers are considered, the “cost of evaluat (no. * 

Is Steal 1 indeed. 

Wr 1 1 i ng obj er t i ves a.nd j terns . Ilir job of sm i Ifvlnp a curriculum 

a course or project in behavioral terns, and writ In* qu stions lo mv.iSut< 
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those objectives, is substantial. One strategy would be for a team of 
teachers and curriculum specialists to work together to define the objectives 
of the project. Staff from similar projects could develop a bank of objec- 
tives and items, and a set of tests could then be tailored to a specific 
project by stratified sampling. A feasible way to expedite this whole 
process is to hold a workshop for several weeks during the summer, with 
curriculum people and/or directors of similar projects grouped together. 

This plan could be considered a desirable modification to Alternative I 
and has been Included in the budget for Alternative 11. 

Collection of all possible raw data . The ideal use of CAM would be 
to monitor all students at each administration, and to do this often, at 
least every two weeks. This would provide the maximum amount of data: 
it would be directly useful to the project (CAM imposes more than average 
structuro on data collection; therefote the date are more valuable); 
teachers would find it e satisfactory substitute for their own test pro- 
grams, and It could be appropriately summarized for state-level collation 
with data from similar projects. 

Choosing an alternative . The three plans for implementation of CAM 
each have particular merit. However, from the point of view of pure evalua- 
tion at the elate level, Alternative III has the most validity built in. 
Ultimately, the cost for state evaluation of state* funded projects is a 
matter of which pocket the money cones out of, rather than from whom. 

No one alternative need be selected on a statewide basis, however. 

The choice could be e matter of excellence of design for evaluation in a 
proposal, as critiqued by the state; or project directors might, express 
a preference: or certain tyoes of projects ms/ be better suited to one 
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plan than another. The most important consideration is that the evaluation 
procedure should serve the project locally as feedback on which to base 
"real time" decisions, and also serve the state-level needs in project 
evaluation. 
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